<html>
<head><meta charset="utf-8"><title>mbe punct unit · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/mbe.20punct.20unit.html">mbe punct unit</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="189658410"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/mbe%20punct%20unit/near/189658410" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/mbe.20punct.20unit.html#189658410">(Mar 04 2020 at 03:45)</a>:</h4>
<p>I just want to confirm what is correct punct token (single / composited) should be used in mbe parsing. Imagine the following program:</p>
<div class="codehilite"><pre><span></span><span class="n">macro_rules</span><span class="o">!</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="p">(</span><span class="w"> </span>-&gt; <span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="s">&quot;-&gt;&quot;</span><span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w">    </span><span class="p">(</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="s">&quot;-&quot;</span><span class="w"> </span><span class="p">};</span><span class="w"></span>

<span class="w">    </span><span class="p">(</span><span class="w"> </span><span class="cp">$first</span>:<span class="nc">tt</span><span class="w"> </span><span class="cp">$($tail</span>:<span class="nc">tt</span><span class="p">)</span><span class="o">+</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="n">m</span><span class="o">!</span><span class="p">(</span><span class="cp">$first</span><span class="p">)</span><span class="w"></span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>

<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">result</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">m</span><span class="o">!</span><span class="p">(</span>-&gt; <span class="nc">a</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="n">c</span><span class="p">);</span><span class="w"></span>
<span class="w">    </span><span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="s">&quot;{}&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">result</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>


<p>I tested it in the <a href="https://play.rust-lang.org/?version=stable&amp;mode=debug&amp;edition=2018&amp;gist=82cdf8e53cb3ca97a0309c7edd824de1" target="_blank" title="https://play.rust-lang.org/?version=stable&amp;mode=debug&amp;edition=2018&amp;gist=82cdf8e53cb3ca97a0309c7edd824de1">rust-playground</a> , the output is <code>-&gt;</code>. I am quite surprised that it use composited token as <code>tt</code>. How about proc-macro token stream ? Does its use composited punct token as well ?</p>



<a name="189675689"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/mbe%20punct%20unit/near/189675689" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/mbe.20punct.20unit.html#189675689">(Mar 04 2020 at 09:57)</a>:</h4>
<p><span class="user-mention" data-user-id="216201">@Edwin Cheng</span> yeah, this is one of the warts of the language. mbe matchers do observe composed tokens.</p>



<a name="189675719"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/mbe%20punct%20unit/near/189675719" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/mbe.20punct.20unit.html#189675719">(Mar 04 2020 at 09:57)</a>:</h4>
<p>Proc macros operate on tokens which carry jointness information with them, so they can do whatever</p>



<a name="189677403"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/mbe%20punct%20unit/near/189677403" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/mbe.20punct.20unit.html#189677403">(Mar 04 2020 at 10:23)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> Do you think we could just handle this case by solely treat <code>$var:tt</code> meta-variable as composited token, without changing the whole mbe parsing infrastructure ?</p>



<a name="189677785"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/mbe%20punct%20unit/near/189677785" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/mbe.20punct.20unit.html#189677785">(Mar 04 2020 at 10:28)</a>:</h4>
<p>sort of. When we match <code>$var:tt</code>, we should capture several tokens</p>



<a name="189677826"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/mbe%20punct%20unit/near/189677826" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/mbe.20punct.20unit.html#189677826">(Mar 04 2020 at 10:29)</a>:</h4>
<p>To be honest, I don't remmeber of the top of my hand how our mbe parsing infra works, but i think it should be fixable with relatively little effort</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>